Improving Medical Student Inpatient Documentation Through Feedback Using a Note Assessment Tool

Introduction Documentation within the Electronic Health Record (EHR) is an essential skill for medical students to succeed in residency and post-residency training. The increased use of medical student progress notes for billable services raises the need for the education and assessment of quality note writing. We hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes. Methods We conducted a retrospective study to review the quality of student inpatient progress notes written before and after structured feedback using the Responsible Electronic Documentation (RED) checklist throughout a third-year internal medicine clerkship. The first intervention group received feedback from clerkship directors in the 2017-2018 academic year and the second intervention group received feedback from ward residents/attendings in the 2018-2019 academic year. Within each intervention group, the total note scores from pre and post-intervention were compared. Results Feedback from clerkship directors yielded a greater increase in students’ total note score from pre to post-intervention compared to ward resident/attending feedback (F(1,255) = 12.84, p < 0.001). Cohen’s d effect size value was greater for the clerkship director feedback arm (d=0.71) compared to the ward resident/attending feedback arm (d=0.24). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in total note scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24). Conclusion Clerkship director feedback led to a greater increase in medical student documentation compared to ward attending/resident feedback. Nonetheless, structured feedback with a note assessment tool, whether from clerkship directors or ward attendings/residents, leads to a significant improvement in medical student documentation. Though there are various methods for providing feedback, educators can use the RED checklist to provide clear guidelines that will facilitate note-writing feedback.


Introduction
Clinical documentation within Electronic Health Records (EHR) plays a key role in patient care and healthcare systems. It creates a record of patient encounters, serves as a device for communication between providers, streamlines coding and billing, and acts as a tool for further safety and quality improvement. Given its fundamental purpose, medical educators agree that documentation within the EHR is an essential skill for students to succeed in residency and post-residency training [1][2][3]. Although both the Association of American Medical Colleges (AAMC) and the Accreditation Council for Graduate Medical Education (ACGME) stress the importance of documentation skills within the EHR [2][3], the majority of note-writing instruction and student experience remains highly variable.
A 2012 survey of clerkship directors by the Alliance for Clinical Education found that only 64% of programs allowed students to access the EHR; of these programs, only two-thirds allowed students to write notes for entry into the EHR [4]. A nationwide survey conducted between 2012 to 2016 by the Liaison Committee on Medical Education (LCME) found similar results, with students entering notes in less than 65% of their third-year clerkships [5]. 1 The limited use of medical student notes in the EHR is partially reflective of previous Centers of Medicare & Medicaid Services (CMS) guidelines restricting the elements of a billable note that could be authored by a medical student. With new revisions released by CMS in 2018, teaching physicians can now use medical student documentation for billable services with appropriate supervision [6]. With these changes, student documentation within the EHR is projected to increase. In doing so, two important questions are raised: (1) "Do medical students write quality notes?" and (2) "can medical students be taught to write quality notes?" Though there is a paucity of literature regarding the objective quality of medical student notes, available studies show that student documentation of patient encounters is often inaccurate, copy-pasted, or filled with "note bloat" [7][8][9][10]. One of the main reasons for the poor quality of medical student notes in past studies may be due to the lack of sufficient and consistent feedback [11]. Effective feedback has been shown to be an important tool for increasing the quality of physician and student clinical performance [12][13][14][15]. However, most of these studies on student note quality involved standardized patients or simulated environments, which may not reflect real patient-doctor encounters in an inpatient or clinical setting [8,[16][17].
We set out to investigate whether student documentation with non-standardized patients could improve with feedback and hypothesized that structured note feedback using a note assessment tool would improve the quality of medical student inpatient progress notes.

Materials And Methods
We At our university, students on their 10-week core Internal Medicine (IM) clerkship rotate at two of five different hospital sites, splitting the clerkship into two inpatient blocks. We conducted the intervention (note feedback) at Week 5 of the 10-week rotation. All students were required to turn in two progress notes for evaluation, one before the intervention and the other after the intervention.
The Responsible Electronic Documentation (RED) checklist was chosen to evaluate the quality of notes. This tool was developed by the Northwestern University Feinberg School of Medicine as a way to evaluate inpatient progress notes (Appendix) [18]. The RED checklist takes minimal time to complete (on average 7-9 minutes), does not require prior knowledge of the patient, discourages copy and pasting from previous notes, and rewards critical reasoning on the assessment/plan portion of the note by the way the points are distributed in the checklist. Other tools, such as PDQI-9, QNOTE, HAPA, and Audit tool (from U of Wisconsin), were considered for use in our study [19][20][21][22]. However, these tools either did not focus on inpatient notes or had mixed results with a large group of students in improving note quality [22][23][24].
We compared the quality of student notes before and after two interventions: feedback by clerkship directors in the 2017-2018 academic year and feedback by ward residents/attendings in the 2018-2019 academic year.

Feedback by clerkship directors
For the first intervention in the 2017-2018 academic year, one of three IM clerkship directors evaluated notes with the RED checklist and then gave feedback during an individual scheduled 15-minute session at mid-rotation Week 5. Interrater reliability was not measured, but clerkship directors had a group training session on how to use the RED checklist to evaluate student notes and determined consensus on how they interpreted the RED checklist. IM clerkship directors had no previous knowledge about the patients being discussed in the note allowing their evaluation to be based solely on the checklist for an accurate, responsible note.

Feedback by ward residents/attendings
For the second intervention in the 2018-2019 academic year, residents and attendings responsible for the same patients as the students provided feedback. They also used the RED checklist to evaluate notes and gave feedback informally during a regular workday at mid-rotation Week 5. The ward residents/attendings did not have any previous training on how to use the RED checklist other than instructions that were written on the checklist itself.
At the end of the rotation (Week 10), both intervention groups had their notes graded using the RED checklist. Students in the first intervention had the second set of notes graded by IM clerkship directors, whereas students in the second intervention had the second set of notes graded by their new ward residents/attendings on their second block of the clerkship.

Data analytic plan
Inclusion criteria included students who had a complete set of data: two inpatient progress notes and two graded RED checklists, from mid and end-of-rotation. In order to examine the efficacy of the interventions in improving students' note writing, we conducted a mixed-design analysis of variance (ANOVA) to determine whether the changes in students' scores on the RED checklist were different depending on the intervention group. A reverse-score square root transformation was performed on the RED checklist data to correct for non-normally distributed data and violation of the assumption of homogeneity of variances for ANOVA. Cohen's d as an effect size was used to measure the magnitude of change in RED checklist scores before and after the intervention. Post-hoc dependent sample t-tests were conducted with each intervention group to determine whether there were significant changes in RED checklist scores before and after the intervention for each group. All analyses were conducted in IBM SPSS 20 (IBM Corp. Armonk, NY).
We received institutional IRB exemption for this study.

Results
For intervention 1 (clerkship director feedback), 124 out of 163 students (76%) met inclusion criteria and were included in the analysis. For intervention 2 (ward resident/attending feedback), 133 out of 155 students (85.8%) met inclusion criteria and were included in the analysis.

Differences in RED checklist scores between pre and post-intervention by feedback from clerkship directors vs. ward resident/attendings
For intervention 1, the mean RED checklist score was 75% pre-intervention compared to 86% postintervention. For intervention 2, the mean RED checklist score was 90% pre-intervention compared to 93% post-intervention.
Results from the mixed-design ANOVAs indicate that the increase in the RED checklist scores was significantly greater from pre to post-intervention for students who received feedback from clerkship directors compared to students who received feedback from ward resident/attendings on the total RED checklist score (F(1,255) = 12.84, p < 0.001). Post-hoc analyses using dependent sample t-tests revealed that there were significant increases in RED checklist scores from pre to post-intervention for both the clerkship director arm (t(123) = 8.26, p < 0.001, d = 0.71) and the ward resident/attending arm (t(132) = 2.85, p = 0.005, d = 0.24). Additionally, the results from the ANOVA indicate that when collapsing across time-points, those in the clerkship director arm were rated significantly lower on average (M=0.90) using the RED checklist than those in the ward resident/attending arm (M=0.75) (p < 0.001). Results and descriptive statistics are summarized in Table 1.

Discussion
Our results found that feedback from clerkship directors yielded a greater increase in students' total note scores from pre to post-intervention and had a larger effect size (d=0.71) compared to ward resident/attending feedback (d=0.24). The results also showed that students' pre-intervention total note scores were lower, on average, when graded by clerkship directors (M=0.75) compared to ward residents/attendings (M=0.90). We propose four potential reasons for the different findings between the two groups:

The Quality of Feedback Between the Two Groups Was Likely Non-Identical
Clerkship directors have much more experience giving feedback to students than residents and most attendings. In the second intervention, while some of the evaluators were attendings, most were residents (68-74% residents, 18-22% by in-house attendings, and 7-8% by unknown evaluators). Clerkship directors are likely more effective teachers with more experience giving feedback compared to residents who lack experience and instruction on giving effective feedback [25][26].

The Difference in Knowledge/Utilization of the RED Checklist by the Two Interventions
More experience with the assessment tool likely contributed to the larger increase of total note scores from pre to post-intervention and the larger effect size in the clerkship director arm. Though the RED checklist is able to be utilized by individuals using it for the first time, those who are unfamiliar with the RED checklist will need to consult a detailed key to grade progress notes [18]. Interrater reliability was not measured, but all clerkship directors met to establish their interpretation of how to grade and create norms. Evaluators of the second intervention did not have this experience before utilizing the RED checklist.

Time for Formal Feedback
Residents are busy and resident teams are often capped at full patient loads. They may be preoccupied with more pressing clinical duties that supersede teaching. We suspect residents could have rushed through grading and giving feedback to students compared to clerkship directors who had a dedicated period of time set aside to provide detailed student feedback.

The Social Aspects Influencing the Inpatient Teams Led to Score Inflation
We suspect a culture of politeness is present to avoid damaging a student's self-esteem with constructive criticism [27]. Face-to-face evaluations of junior medical students resulted in grade inflation in one study [28]. Though these factors were likely present in both interventions, we hypothesize that there was a greater influence in the second intervention consisting of mostly resident evaluators due to the "social desirability bias" [29]. Since residents spend significantly more time with medical students throughout a clerkship and are near peers, they may be more inclined to give higher scores in an effort to build and maintain strong, interpersonal relationships with students.
Although clerkship director feedback was more effective than ward resident/attending feedback, our posthoc analyses showed that the quality of medical student notes, as assessed by the RED checklist, significantly improved regardless of the source of feedback.
Our study has several strengths. Unlike past interventions, our study assessed real non-standardized inpatient progress notes written on wards. We had a large sample size consisting of 257 third-year medical students: 124 students (76%) in the 2017-2018 academic year and 133 students (85.8%) in the 2018-2019 academic year. A total of three different EHR systems (Epic, computerized patient record system (CPRS), and PowerChart) were used at the five different hospitals where students rotated, thus making these results generalizable to a variety of hospital systems.
This study was restricted to the Internal Medicine clerkship, and it is unclear if this intervention will improve student notes in other disciplines. Another limitation was the fact that interrater reliability was not measured. The greatest limitation of the study was a lack of a control group. We felt the feedback was critical to a student's progress thus our study did not implement a control group that did not receive feedback. Future studies are needed to compare the note quality with feedback vs. no feedback. Further analysis comparing note quality scores on the same note when graded by a novice grader compared to an experienced grader is currently underway.
When attendings and residents give targeted feedback on notes, the quality improves. As obtaining feedback has been shown to be an integral part of medical education, the RED checklist can also help initiate the process of asking for and obtaining quality feedback from residents and faculty members. Many medical educators struggle to find time for providing structured quality feedback to trainees and lack tools to facilitate structured feedback. The RED checklist allows clinicians and residents to evaluate notes in seven to nine minutes without the need for any background information on patients. The use of various feedback methods and the nature of the tool allow the easy implementation of any clerkship or program. The RED checklist can also help provide students with clear expectations and guidelines on how to write a wellstructured note. Furthermore, teaching hospitals nationwide should consider incorporating tools similar to the RED checklist to facilitate note feedback, as they begin to look for ways to evaluate the appropriateness of clinical notes written by students.

Conclusions
In today's healthcare environment, clinical documentation plays a major role in patient care. Having accurate and clinically reasoned documentation is key for providing quality medical care. Our results showed significant improvement in student note quality as measured by the RED checklist after implementation of note feedback by either clerkship directors (intervention 1) or residents/attendings (intervention 2). As we continue to search for an effective and time-efficient method to teach note writing and improve student note quality, medical schools can consider implementing the RED checklist into their curriculum to facilitate note feedback and improve medical student documentation.

Objective (cont.)
The data portion of the note contains: No A summary statement is a snapshot overview distinct from the individual problem list.
No = No or inadequate summary statement is included. Yes = Summary statement is included.
9. The summary statement is different from the previous day's statement.
Any change is adequate. Even correcting a typo from the prior day reflects that the summary statement has been reviewed.
No = The summary statement is identical to the previous day's. Yes = The summary statement is different. N/A = There is no summary statement.
Assessment and Plan (cont.) The assessment and plan meet these criteria: No

A problem-based assessment is included
The assessment is organized by problem, not listed by organ system (i.e., not cardiovascular, pulmonary, infectious disease, genitourinary, etc).

No = A problem-based assessment is not included. Yes = A problem-based assessment is included.
Mark on the scale as defined in key: 0 1 2 N/A 12. The status of each problem is described.
Look at ALL items in the A/P. All problems should be described as improving, worsening, persistent, stable, resolved, new, or inactive.
Include all problems except structural, hospital-specific paragraphs such as code status, dispo, contact info, prophylaxis. The terms acute or chronic do not fulfill the criteria for this item. Status of lab values (e.g., creatinine improving) do not fulfill criteria for this item unless the problem is the lab abnormality itself (e.g., hypokalemia worsening). 14. Interpretation of studies is included. 15. Problems are written as diagnoses or accompanied by differentials.
Look only at the first 3 items. The header for each paragraph should either be a disease or a symptom/abnormality with a differential (at least one possible diagnosis) easily identified under it. A diagnosed lab abnormality (e.g., hyponatremia) should include a cause or differential. Exclude structural, hospital-specific paragraphs such as code status, dispo, contact info, prophylaxis. 0 = None of the first 3 items in the A/P is labeled as a disease (or symptom/abnormality with a differential). 1 = At least one of the first 3 items of the A/P is labeled as diseases (or symptoms/abnormalities with a differential). 2 = All of the first 3 items in the A/P are labeled as diseases (or symptoms/abnormalities with a differential).
16. Active problems are accompanied by clinical reasoning.
Look only at the first 3 items. Clinical reasoning is defined as any stated reason for either a plan or a diagnosis or assessment.

Summary -Faculty to complete
A good progress note is TRUTHFUL, REASONED, UPDATED, AND SUCCINCT. Please comment on characteristics of this note that fulfill or lack these characteristics This note appears TRUTHFUL. Portions do not contradict one another or the prior note.
This note is REASONED. It reflects rational clinical thought processes by the author.
This note is UPDATED from the previous day's note. It attempts to communicate the current state of the patient.
This note is SUCCINCT. It is concise and easy to read.

TABLE 2: Responsible Electronic Documentation (RED) Checklist
Source: [18] Additional Information Disclosures Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.